DOCUMENT RESUME 



ED 225 178 



CS 207 330 



AUTHOR 
TITLE 
PUB DATE 
NOTE 

PUB TYPE 



EDftS PRICE 
DESCRIPTORS 



IDENTIFIERS 



ABSTRACT 



Donlan, Dan 

The Reliability of the Methodology Inventory, 

[81] . 

25p. 

Tests/Evaluation Instruments (160) — Reports - 
ResearcD^Technical ( 143 ) 

MP01/PC01 Plus Postage • 

^Attitude Measures; Elementary Secondary Education; 
Evaluation Criteria; Faculty Development; *Teacher 
attitude? ; * Teacher Behavior; Teachers; *Test 
Reliability; Test Validity; *Writing Instruction; 
♦Writing Research > 
♦Methodology Inventory 



The Methodology Inventory is a 76-item 4-point * 
measure divided into eight scales designed to determine what 
behaviors a teacher engages in while teaching. The eight scales are 
stimuli for writing, prewriting, writing, revising/editing, 
evaluation, language development, varieties of discourse, and 
publication of written work. The instrument was administered over a 
3-year period to 104 English language arts teachers who were 
participants in various writing project summer workshops. An item 
analysis of the results was performed to determine (1) the 
reliability of the total instrument, (2) tt*e>reliabili ty of items 
within each scale, and (3) the intercorrelitfion among scores on the 
scales. Results showed that the instrument as a whole had a high 
reliability coefficient and that each of its eight scales was 
internally consistent. Low to moderate intercorrelat ions among the 
scales suggested that there was a commonality that cut across the 
scales; yet the commonality was not so large as to suggest that the 
scales were not somewhat independent of each other. (Extensive tables 
of data and a copy of the instrument are appended.) (FL) 
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With the proliferation of staff development projects in English, 
particularly composition, it becomes necessary to evaluate the impact 
of these piograms on -the teachers who participate in them- One way is 
by assessing teacher attitudes toward instruction. One yay of as- 
sessing attitudes is to determine what behaviors a teacher engages in ^ 
^ while teaching writing- Such an inventory of behavior w.as developed 
by Donlan (1979),. an instrument called The Methodology Inventory . The 
Inventory was originally a 70-item A-point measure subdivided into 
seven scales: (1) stimuli for writing, (2) prewriting, (3) writing, 
(A) revising/editing, (5) evaluation, (6) language development, and 
(7) miscellaneous. The' instrument was sent to ^panel'of judges, ex- 
perts in the teaching of composition, who were to respond to 5 
questions: 

k Were the definitions given with the A-point 'scale, adequate? 
' 2. Taken together, were the categories a comprehensive assess- * 
ment of what teachers do by way of teaching writing? • 
3. Was each category complete? ^ 
* A. What items should be deleted or added? 

5. Should the items remain in categories or be scrambled? 
Based on the judges' suggestions, (1) 'items were dropped, (2) items 
were added, (3) items were transferred to other categories, and the 



scale called "miscellaneous" was replaced by 2 scales— "Varieties of 
Discourse" and "Publication of Written Work." The resulting inventory 
contained 76 itetas, subdivided fnto 8 scales. 

Haviji| thus achieved face validity, "the instrument was 
administered over a. period of three years to 104 English language arts 
teachers. * These were teachers who had been admitted to local writing 
project summer workshops. An item analysis was performed to determine 
(1) the reliability of the total instrument, (2) the reliability of 
items within each scale; and (3) the intercorrelation am6ng scores on 
the Inventory's eight scales* * 
Reliabilty of the total scale 

Based on the sample of 104 teachers, mean'scores and^standard de- 
viations were obtained and test reliability was calculated for each of 
the 76 items. Three estimates of reliability w$re administered. The 
Cronbach Alpha yielded an overall reliability of .93. Hoyt yielded a 
reliability of .90 without the effects of items removed and .93 with 
the effects of items removed. Testing for split-half reliability 
(Helmstadter) corrected with Spearman-Brown yielded .95 reliability; , 
when corrected with Guttmans, a .95 reliability. 
Th e r eliability of items within each scale (internal consistency) 

A mean score for each item in each scale was correlated with the 
total scale minus the item, using Pearson's correlation coefficient. 
Tattle 1 lists the items in each one of these scales. Items with 



Insert Table .1 abdut here 
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of these -scales. Jtems. with correlations at or below .20 were 
eliminated from the scale (after Schuessler, Gere, and Abbott, 1981). 

. Afte& the' determination of the correlation of each item within a 
'scale to. the total scale minus the item the reliability of each scale 
was estimated. Table 2 shows the number of items, means, Standard 



Insert Table 2 about here 



deviations', coefficient a*lpha and split T half (corrected) reliability 
coefficient* for eight scales of 'the Methodology Inventory. 

As Table 2 indicates, each of tsjie 8 'scales has a reasonably high 
reliability, even scales with small' numbers of ^tems (e.g., Scales 3, 

7, and 8), • 
Intercorrelations among-the scales 

Scores on each of, these eight scales were injercorrelated and are 

i 

shown in Table 3. This' was to determine t^e degree of independence 



Insert Table 3 about here 



t 

the scales had, 



As shown in Table 3, the intercorrelat ions^range between .13 and 
.65, a range from low to moderate. The lowest intercorrelat ion exists 
between stimuli for writing and'revising/editing (.13). Since the 
square of the intercorrelation equals the percentage of scared vari- 
ance, the shared variance is only 1.62. In effect, the process of 
stimulating the student to write is fairly independent of the process 



of revising/editing as measured by /these scales. On the other hand, 
the highest . intercorrelation is between revising/editing and 
evaluation < .65); this interctfrrelation is to be expected , since 
students frequently revise papers after they have ^een evaluated. 
Yet7 even the shared variance between these two* categories is anly 
42%, the moderate intercorrelatfons suggest two aspects of the 
Methodology Inventory: (1) The Methodology Inventory seems to be % 
measuring'a common ^lemeat in the teaching of composition that cuts 
across the eight scales; but (2) the limited shared variance between 
scales suggests a high degree of independence of th£ eight scales. As 
a result, the Methodology Inventory could be considered a multi- 
dimemsdlonal measure. 

\ J-^ Conclusion 
Based on the analysis of data from a sample of 104 teachers, the 
instrument as a whole has a high reliability coefficient. In ad- 
dition, each of the instrument's 8 scales is internally consistent. 
Also, the low to moderate intercorrelations among the scales suggest 
there is a commonality that cuts across the eight scales; yet' this 
commonality is not so large'as to ?uggest that the scales are not 
s6mewhat independent of one another. ■ As the teaching of composition 
is a multidimensional process^ the Methodology Inventory , as has been 
shown, can reliably measure these dimensions. 
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Items in eac,h sub-category, Mean (M) 'and Standard Deviation (SD) 
of item responses, and the Correlation (r) 'be.tueen the item 
and the scale excluding^the item (*Item subsequent!? deleted).- 



Scale 1. Stimuli for Writing 




2.33 .99 .27 1. Before students begin writing, I show a film- 

to give them some ideas y \ 

2.34 .88- ' .57 2. Before students begin writing, I play music 

or spoken recording to give them some- ideas. 

3 # 05 .92 .39 3. Before students begin writing, I provide 

concrete objects (realia) to giye them some 
jf* ideas. 

2.15 ' 1.11 .42 4. Before students begin writing, I engage the 

class in creative dramatics. 



3,03 .89 .54 5* Before students begin writing, 1 show them 

pictures to give them ideas. 

2.11 1.05 .40 6. Before students beg4.'n writing, I take them 

outside of the 'classroom tq make observations 
of the world around them. * 4 

1.86 .83 ..32 7i Before students begin writing; I have thep 

watch television to get ideas. \ 

l ; '40 .70 .30 8. Before students begin writing, I have th£m 

listen to the radio t6 get ideas. . \ 



1,64 ' .77 *45 9. Before students begin writing, I have them 

listen to guest speakers who can provide them • \ 

with ideas for waiting. ' 

3.41 .87 .35 10. Before students begin writing, I have them • 

* i;ead stories, poetry, or plays to get ideas 
A for writing. » , 

3.01 1.00 .29 11. Before students begin writings 1 have them 

/read "nonfiction (e.g., essays, articles, 
newspapers, magazines) to get ideas for 
writing. 

1.85 .95 *51 ' 12 » Before students begin writing, 1 have them 

engage in game playing. 
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2.38 



Ta^le 1 '(Cent.) 





i 

* 




• 

M 


. _. „..- — . ■ i ■ — - — — : — — — 

Scale 2. Prewriting 

\ > ■ 


M 


SD 


r 




\ Item , 1 ^ 


J 

•> 






— * 


Befor£ students begin to write, I eNigage them 
in self-§vareness exercises, such as com- 
piling a sensory checklist. 


3. 1U 




97 


0 

z, • 


Rpfore students begin to write, I have th^ni 
compile word lists tharj. they might draw on. 
while wrirting. 

✓ 


2.77 


1.10 


.36 


3. 


Before students begin to write, 1 have ttiem- 

engage in outlining or comparable^org^ni- 

zational schemes.. , . 
✓ * 



3. 48* .76 .41 4. Before students begin to write, I engage the 

' class as a whole in preparatory discussion of 
a directive nature. 

2.95, .94 -39 5- ' Before students begin to write, I engage the 

' class* as a whole in prepartory discussion of 
a nondirective nature. 



# 97 .48 6. Before students begin to write,' I have them 
meet in small groups in preparatory dis- 
cussion. 



2.74 1.07 .27 > 7. Before Students begin to write, I have in- 
dividual conferences with some of them in ' 
which we discuss the assignment. 



r 



2.49' 1.25 ' .35 8. Before students begin to write a formal 1 

\V ^ assignment, I engage them in diary, jorurnal , 

/• \ or log writing from which J>hey can draw 

| \ • ^ ' ideas. , , 

9# 0n a *8 iven assignment-I prpvide students 

^«**u>j**& . w ^ t ^ a variety" of topics from which to 

choo'se. 

2.69 1.01 .44 10. On a given assignment I provide students with 

i a variety of writing types they may use. 



8 
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• , Scale 3« Writing. ^ 

; > ! — - : — T 

M SD r Item , , / 



a- 



*3.68 .56 ".13 1. I have m^ students write in class' under my 

supervision. 

* 

3.76 .62 .33 2* While students a*Q writing, I move around the 

classroom volunteering assistance. 

\ ' > 

2.37 -1.12 .33 3. v While students are writing, I schedule in- 
termittent discussions about work in 
^progress, engagirig the entire class in the 
^ discission. * t 

2.12 1.1*5^ .59 4'. While students are writing, I schedule 

intermittent small group discussions about 
work inj>rogressi - 

2.77 1.06 ' .46 5. While studefnts are writing-, I have them share 

work in progress by reviewing* each other's 
papers. . 1 

2.30 ,1,01 *30 6. I write the assignments right alpng with the 

students. / * 

1.88 1.04 .39 7, I have students put writing in progress on 

the bo£u:d* on overhead projector, or on 
opaque 1 projector for entire\ class oj small 
group discussion. ^ . \ 

2.21 1.08 .h% 8. I dupl&a/e students' writing in progress for 

» entire cLiss or small group discussion. 

' 2.49 1.01* .33 9. I have my students write for. non-school 

purp9ses. 




c 



( 



.°47 



Table' 1 (Co*>c 



Scale A. Revising^Erfiti^g 



Item 



f .58 1. I require a second or otherwise final draft 
of each student. , 

.AO 2. I H>ad the. firs* (rough) dctfv bsrff re. each • 
student revises It. 

.36 3. I correct and othery^ise evaluate tfie first % ' 
(tough) draft before Vach student Revises it. 

.42 4. I have students evaluate'tJieir own roqgh , 
drafts without help from Me or othe*f 
students. 

•5. I have students revise thoir ovn rough drafts 
i nonfunction With student" ecM ring Y 
cc*w»itiees, small fcroup psf>ir e^al foti/>n* o*r 
' soi»e other tor*! of student to ^sfusJanf 
confa*encing. - 

.53 *6. I require x\\ix students subwr Their rough 
draft(s) along with their finished dr*ft. 
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Scale 5. Evaluation * 
• ^ 






• M • 


SD 


r 







3.58 .66 

'2.01 .99 

2.53 1.13 

' 3.58 .78 

3. 34 .99 



2.72 1.04 



.27 



.31 



.25 



.31 



.-58 



.43 



•1. Students are evaluated on the basis of 
criteria I have established. ^ 

2. Students are evaluated on the basis of 
/ criteria they have established. t 

3. Students are evaluated on the basis of 
criteria arrived at jointly between them and ^ 
me • 

4. I write a general comment to £he student 
about the paper; the comment appears either 
at the beginning or at the end of the paper. 

5. To the student, I make marginal comments that 
refer ta spejcific aspects of a paper — a* 
phrase, a sentence, a paragraph, anxidea, • 

. etc., 

6. I rewrite parts of student papers'; speci- 
fically, I show-students how they might have 
written a phrase, a sentence, or a paragraph 
mdre effectively* 







2.77 


.97 


".24 


7. 


I indicate errors without correcting them. 






*1.60 


.85 


-.08 


8. 


I neither indicate nor correct errors. 






2.00 


1.10 


.35 


9. 


I use small groups of students to evaluate 
writing. 






2.91 


1.18 


.54 


10. 


I assign grades to Vriting assignments. 






2.50 


1.21 


.24 


11. 


*L assign a single grade which represents a 
combined evaluation *pf content aq^f^rm. 






- 2.31 


-1.26 


.47 


12. 


0 

I assigh a split grade; one grade for con*- 
tent, the other for form. 






1.63 


-.88 


.28 


13: 


I grade ^on content only. 






1.50 


.78 


.28 




I grade on form only. 

* t 


1 




1.90 


.84 


.35 

4 


15. 


r 

I let students grade themselves. 
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Table 1 (Cont.) 



Scale 5. Evaluation (Cont,) 



M 



SD 



It$m 



1.88 .96 J .56 16. I let students grade each other's writing 
| . * assi8pmer|bs^ 

3^32- .98 i § .29 17. I have students keep cumulative folders 

containing their writing. ■ . 

2.56 1.^ ' .54 18. I help students with writing assignments , 

from other classes. - * 

1.85' .87 .34 '19. I assign writing sttictly for purposes of 
» diagnosis, < ✓ • 



J . 
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Scale 6» Language Development 



M SD r ' Item 



3.25 .94 .31 1. I teach formal grammsr between writing 

assignments* 

'3.50 ;74 .47, - 2. X teach .usage between writing assignments, 

concentrating on those problems studepts 
ha\^e exhibited on the previous assignment. 

*2.61 .93 .18 3. I teach usage between writing assignments,^ 

but not directly related to usage problems 
; students have exhibited on previous assign- 

ed ments. 

2.85 .99 .52 -4. Between writing assignments, 1 have students 

engage in sentence-combining exercises. 

1.89 1.03 .47 5. Between writing assignments, I have students 

generate sentences that model J:hose of re- 
% v spected writers. 

> 2.70 1.00 .61 6. Between writing assignments, L.have students 

engage" in exercises involving semantics: 
connotation, denotation, figurative* language. 

2.27 1.05 .53 7. Between writing assignments, I have Students 

engage in paraphrasing other writer's 
language. 



2.04 .97 -46 8. Between writing assignments, I have students 

model- sentences at>d paragraphs that other 
^jl . writers have created. 

±*Z * 
*3.29 .80 .16 9* Between writing assignments I engage the 

students in oral language activities e-g. , 
dramatics, discussion, problem solving. 

* 2.3J" 1.03 .45 10. Between assignments, I perform linguistic " 

analysis of student sentences, i.e., for 
syntax, semantics , rhetoric. 



< 
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Scale 7. Varieties of Discourse 



M 


SD 


r 

i 










It em 






3.18 


.86 


.49 


1. 


I 


have 


students 


engage 


in 


narrative writing. 


> 

3.35 


.88 


.46 


2. 


I 


have 


students 


engage 


in 


expository writing. 


3.01 


.98 


.44 


3. 


I 


?ave 


.students 


engage 


in 


report writing. 


2. 54 


1.07 


.53 


4. 


I 


have 


students 


engage 


in 


argumentative and 



persuasive writing. 

2.87 1-01 -37 5. I have students write poetry. 

2.41 1.11 *53 6- I have students write drama: e.g., skits, 

short plays. 



Scale 8. Publication of Written Work 



M SD 



Item 



3.21 1.04 .26 1. I place student writing on display in the 

classroom. 

* |^24 1.26 .56 2. I publish classroom anthologies of student 

writing. 

2.31 1.28 .46 3. I submit student writing to school literary 

magazines. 

« 

2.02 1.7 «41 4. I # submit student writing to local, state, • 

or national writing contests. 
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Table 2 



Number of items (N) , Means (M), and Standard Deviations (SD), 
coefficient alpha and split-half (corrected) reliability 
coefficients for eight scales of the methodology 

inventory. t 





Scale Name 


N 


M 


SD 


Alpna 


C— U 

. on 


1. 


Stimuli for Writing 


12 


28.2 


5.8 


.76 


.81 


2. 


Prewriting 


10 


28.4 


5.1 


.69 


.73 


3. 


Writing 


9 


25.8 


4.8 


.70 


.81 


4. 


Revising/Editing 


6 


17.5 


4.0 


.72 


.78 


5. 


Evaluation 


19 


46.5 


8.4 


-.78 


.85 


6. 


Language Development 


10 


26.8 


5.3 . 


.75 


.78 




Varieties of Discourse 


6 


17.4 


3.9 


.73 


.71 


8. 


Publication 

N 


4 


9.8 


3.2 


.64 . 


.76 



6 

i 
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Table 3 



Intercorrelation among scores on eight scales 
of the Methodology Inventory 



A 


5 


6 


7 


1 

.13 


.29 


.43 


.47 


.42 


.56 


.48 


.56 


.48 


.55 


.37 


.55 




.65 


.40 






/ 


.44 






1 




.46 



\ 
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Methodology, Inventory* 
(as revised May 20, 1981 ) 



INSTRUCTIONS : In preparation for your possible participation in the Inland Area 
Writing Project, <the organizational committee would like to plan a program based 
upon your experiences this past year in the teaching of written composition. Please 
read each Item carefully and check the appropriate box: A (I performed this 
activity frequently tfcis^past year), B (I performed this activity occasionally this 
past year), C (I performed this activity infrequently this past year), and 0 (I 
didn't perform this activity this past year). 



Stimuli for Writing A B C D 



< k I. 


Before students begin writing, I show a 
film to give them some Ideas. 






> 




2. 


Befpre students begin writing, I play 
music or 'spoken recordings to give ttjem 
some ideas/ ( 










3. 


Before students begin writing, I provide 
concrete objects, (real ia) to m give 
them some ideas. 










4. 


Before students begin writing, I engage 
the class in creative dramatics. , - 










5. 


Before students begin writing, 1 show 
them pictures to give them ideas. 




L 






6. 


r 

Before students begin writing, I take 
them outside of the classroom to make 
observations of the world around them 










7. 


Before students begin writing, I hav£ 
them watch television to get ideas. 


■ 








A. 


Before students begin writing, I have 
them listen to the radio to get ideas. 










9. 


Before students begin writing, I have 
them listen to guest speakers who can 
provide them with ideas for writing. 








« 


10. 


Before students* begjn* writing*, I have 
them read stories, poetry, or plays 
to get ideas for writing. 










* 11. 


Before students begin writing, I have 
them read nonf iction , (e.g., essays, 
articles, newspapers, magazines) to 
get ideas for writing. 


• 








- 12. 


Before students* begin writing, I have 
them engage in game playing. 











♦Reproduce only-.wit'h written permission of author, Oah Donlan 
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Prewritin q 

1. Before students begin to write, I have 
them compile word lists that they might 
draw on while writing. 

2. Before students begin to write, I have 
them engage in outlining or comparable 
organizational schemes. 

3. Before students begin to write, I engage 
theclass as a whole in preparatory 
discussion of a directive nature. 

4. Before students begin to write, I engage 
the Qlass as* a whole in preparatory 
discussion of a nondirective nature. 

5. Before students begin to write, I have 
them meet in small groups in prepara- 
tory discussion. 

c 

6. Before students begin to write, I have 
individual conferences with some of 
them in which we discuss the assignment. 

7. 8efore students begin to write a formal 
assignment, I engage them in diary, 
journal, or log writing from which they 
can draw ideas. 

8. On a given assignment I provide 'students 
with a variety of topics from which to 
choose. 

9. On a given assignment I provide students 
with a variety of writing types they 
may use. 



1 . Whil£ students are writing, I move 
around the classroom volunteering 
assistance. 

Z r While students are writing, I schedule 
^ intermittent discussions about work in 

progress, engaging the entire class in 

the discussion. 

3. While students are writing, I schedule 
intermittent small group discussions 
about work in progress. 

4. While students are writing, I have them 
share work in progress by reviewing 
each others papers. 

5. I write the assignments right along 
with the students. 

6. I have students put writing in progress 
on the board, on overhead projector, or 
on opaque projector for entire class or 
small group discussion. 

7. I duplicate students 1 writing in pro- 
gress for entire class or small group 
discussion. 



8, I have my students write for non-school 
purposes. 



Revising/Editing 

1 . I require a second or otherwise 
final draft of each student, 

2, I read the first (rough) draft 
before each student revises it, 

3, I correct and otherwise evaluate, 
the first (rough) draft before 
each student revises it, 

4, I have students evaluate their 
own rough drafts without help 
from me or other students. 

5. I have students revise their own 
rough drafts in conjunction with 
student editing committees, small 
group peer evaluation, or some other 
from of student^to-student confer- 
encing, s~ 

6. I require that students submit 
their rough draft(s) along with 
their finished draft. 



Evaluation ( 

1. Students' $re evaluated on the 

basis of criteria I have established. 

/»• 

2. Students are evaluated on the basis 
of criteria they^have established. 

3. Students are evaluated on the basis 
of criteria arrived at jointly 
between them and me. 

4. I write a general comment to the 
student aboyt the paper; the comm6nt 
appears either at the beginning or 
at the end of the paper. 

5. To the student, I make marginal 
comments that refer to specific aspects 
of a paper—a phrase, a sentence, 

, a paragraph, etc. 

6. I rewrite parts of student papers; 
specifically, I show students how 
they might have written, a phrase, a 
sentence or a paragraph more 
effectively. 

7. I indicate errors without 
correcting them. 

8. I use sm^ll groups of students 
to evaluate writing. 

9. I assign grades to writing 
assignments. 

10. I assign a single grade which 
represents a combined evaluation 
of content and form. 

11. I assign a split grade: one grade 
for content, the other for form. 

12. I grade on content only. 

13. I grade on form only. 

14. I let students grade themselves. 



15. I let students grade each other 1 ** 
writing assignments. 



Evaluation (cont.) 
) 6. I have students keep cumulative 
folders containing their writing 

f7. I help students with writing 
assignments from other classes. 

18. I assign writing strictly for 
purposes of diagnosis. 



/ 

i 



■ : ) ■ 

/ 

< 

t 

Language Development 

1. I teach fontial grammar between 

writing assignments, 
# 

2. I teach usage between writing assign- 
ments, concentrating on those problems 
students have exhibited on the previous 
assignment. 

w , 

3. Between writing assignments, I have 
students engage in sentence-combining 
exercises. 

4. Between writing assignments, I have 
students generate sentences that model 
those of respected writers. 

5. Between writing assignments,* I have 
students engage in exercises involving 
semantics: connotation, denotation, 
figurative language.^^ 

6. Between writing assignments, I have 
students engage in paraphrasing other 
writers 1 language. 

7. Between writing assignments, I, have 
students model sentences and para- , 
graphs 'that other writers have created. 

8. Between assignments, I perform linguis- 
tic analysis of student sentences, ne., 
for syntax, semantics, rhetoric. 

i 



J 



Varieties of Discourse 

\. I have students engage in 
^narrative writing, 

2, 1 have students engage in 
expository writing, 

3, I rajy^" students engage in 
rejroVt writing, 

4, I have\ students engage in 
argumentative and persua- 
sive writing, 

5, I have students write -poetry, 

6, I have students write drama: 
% e.g;, skitsAshort plays. 



\ 
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Publication of Written Work 

1. I place student writing on display 
in the classroom. 

2. I publish classroom anthologies of 
student writing. 

3. I submit student writing to school 
literary magazines, 

4. I submit student writing,to local, 
state, or national writing contests. 
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